This disclosure relates generally to resource management in a data processing system and more specifically to managing resource pools for deadlock avoidance in the data processing system.
Transaction processing applications, including Web applications, commonly connect to a database using a technique referred to as connection pooling, which caches and serially reuses physical connections implemented using Java Database Connectivity (JDBC) application programming interfaces. JDBC is a Java-based data access technology available from Oracle Corporation (Oracle® and Java® are registered trademarks of Oracle and/or its affiliates.) providing an application programming interface the Java programming language that defines how a client may access a database. JDBC is directed toward supporting various relational databases.
Connection pooling is a typically well-known technique for avoiding processing costs of establishing a network connection, authenticating an application with a target database, and allocating and freeing agent resources of the target database for each database session. Connection pooling is also used with resource managers other than databases, for similar reasons.
Application servers typically provide a managed database connection pooling facility that leverages a vendor provided database JDBC pooled connection connection factory. The connection pools have configuration parameters that typically include a maximum size pool parameter. The purpose of the maximum size pool parameter is to apply a limit to the connection resources that the application can consume. When an application thread requests a connection (and the request cannot be satisfied by creating a new handle to an active sharable connection with the correct context), a connection pool manager decides a course of action. The connection pool manager determines whether an idle connection is available in the pool, and responsive to a determination that an idle connection is available reuse the idle connection. Responsive to a determination an idle connection is not available and when the current pool size is less than the maximum size, allocate a new connection and return the new connection to the requester. However when an idle connection is not available and the current pool size is at the maximum size, wait for a busy connection to become idle and reuse the idle connection.
However, some applications concurrently request more than one connection from a pool to complete a transaction. Concurrent requests may occur due to a programming error, but also may be legitimate requests such as requests which occur when the application needs to perform synchronous tasks involving database writes that must commit independently of the unit of work whose execution triggered the task. For example, primary key generation and audit logging are two such tasks associated with transactions of a web based commerce application, because for correctness or performance reasons the tasks must not rollback when the original transaction fails. To accomplish the tasks, the commerce application suspends the global transaction and acquires a second connection with a distinct transactional context, performs the task, commits, releases the second connection and resumes the suspended global transaction.
Using the example, a request for a second legitimate connection from the connection pool increases the possibility of resource exhaustion deadlock. Deadlock can occur when the application requires more than one concurrent connection per thread, and the database connection pool is not large enough to serve the number of threads requesting connections. In an example scenario, each application thread requires two concurrent database connections and the number of threads is currently equal to the maximum connection pool size. Deadlock can occur when each thread has a first database connection, and all connections are in use and each thread is waiting for a second database connection, and no connections would become available because all threads are blocked.
A deadlock situation may be broken by a connection wait timeout. Application servers typically provide a configurable timeout, with a default number of minutes, but use of the timeout is not a practical solution in this scenario because a thread that eventually incurs that timeout typically also holds database locks and other resources allocated to the suspended global transactions associated with a first database connection. In a busy server, holding locks even for only a few extra seconds can severely impact the progress of other threads.
A conventional solution is to avoid deadlock by configuring the connection pool maximum size parameter to be greater than the total number of threads that can allocate connections. Using a maximum size parameter set sufficiently large, implies when every thread has acquired a first connection, connections will remain in the pool to satisfy second requests. Using this example solution is extremely pessimistic and typically leads to other problems, particularly with respect to resource usage.
A calculated maximum pool size may be more than 20 times a normal connection pool population when the application is running under design load. When an attempt to populate the pool to a maximum while maintaining connections alive although 80% or more of the connections are idle, a significant quantity of application server and database resources will be consumed. Additionally common firewall security policies in the datacenter that mandate killing idle connections will have to be circumvented.
Another conventional solution only populates the pool as needed. In this case a minor increase in load or degradation in throughput typically causes additional connections to be opened rapidly until the pool maximum is reached. However, in this case the pool is ineffective in achieving a main goal of avoiding a cost of opening connections, and extra load from creating new connections typically results in a connection storm where requests have to increasingly enter a queue which means new requests arriving at the application server activate more threads thereby requiring ever more new connections.